This transformation naturally implements a parameterization for the relaxation of permutation matrices, allowing for gradient-based optimization of problems involving permutations.

HR-based orthogonal fine-tuning is equivalent to an adaptive low-rank adaptation.

In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works.

However, only the smooth setting was considered. To close this gap, we introduce and analyze O-ZD, the first structured finite-difference algorithm for non-smooth black-box optimization.

Neural Information Processing Systems http://nips.cc/

In theory, the techniques can speed up matrix operations, however,inpractice, theyare not fastenough.